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Section  2:  Objectives  and  Overall  Approach 


Goal:  The  main  goal  of  this  project  was  to  investigate  algorithmic  approaches  to  create 
scalable,  robust,  multi-scale,  and  effective  urban  surveillance  and  crowd  control  strategies 
using  UAVs  and  UGVs. 

Overview  of  Approach:  In  order  to  achieve  the  goal,  a  comprehensive  planning  and 
control  framework  was  designed  and  developed  based  on  dynamic-data-driven,  adaptive 
multi-scale  simulation  (DDDAMS)  (see  Figure  1),  where  dynamic  data  is  incorporated  into 
simulation,  simulation  steers  the  measurement  process  for  data  update  and  system  control, 
and  an  appropriate  level  of  simulation  fidelity  is  selected  based  on  the  time  constraints  for 
evaluating  alternative  control  policies  using  simulation.  In  this  project,  the  DDDAMS 
framework  that  was  developed  and  demonstrated  by  PI  Son’s  group  for  the  case  of 
extended  manufacturing  enterprise  have  been  leveraged  and  further  developed  to  address 
scalable,  robust,  multi-scale,  and  effective  urban  surveillance  and  crowd  control  via  UAVs 
and  UGVs.  The  research  outcomes  (e.g.  DDDAMS  framework,  information  aggregation 
framework,  underlying  algorithms,  and  experiments  involving  simulation-based  testbed) 
developed  in  this  project  have  been  published  in  multiple  journals  and  conference 
proceedings  (see  Section  6  for  a  complete  list  of  publications).  More  details  on  the 
proposed  planning  and  control  framework  will  be  explained  in  Section  3. 

Detailed  Objectives  (Tasks):  The  proposed  approach  is  composed  of  the  following  three 
major  tasks: 

•  Development  and  refinement  of  coherent  planning  and  control  framework  (DDDAMS) 
for  effective  and  efficient  control  of  UAV/UGVs 

•  Development  of  algorithms  to  improve  performance  of  coordinated  UAV  and  UGVs 
in  tracking  and  controlling  human  crowd 

•  Development  of  a  hardware-in-the-loop  simulation/control  framework  and  testbed  for 
crowd  control 

The  progress  made  to  date  for  each  of  these  tasks  is  described  in  Section  3. 


Section  3:  Summary  of  the  Efforts 

This  section  provides  details  on  the  efforts  that  were  made  for  three  major  tasks  defined  in 
Section  2. 

3.1.  Task  1:  Development  and  refinement  of  coherent  planning  and  control 
framework  (DDDAMS)  for  effective  and  efficient  control  of  UAV/UGVs 

Figure  1  shows  an  overview  of  the  proposed  DDDAMS-based  planning  and  control 
framework  for  surveillance  and  crowd  control  via  UAVs  and  UGVs  that  was  developed 
and  refined  in  this  project.  The  major  components  of  the  framework  include  1)  real  system 


(UAVs,  UGVs,  human  crowd,  and  environment),  2)  integrated  planner,  3)  integrated 
controller,  and  4)  decision  module  for  DDDAMS.  The  proposed  framework  was  aimed  to 
enhance  the  surveillance  and  crowd  control  capability  of  UAVs  and  UGVs  in  terms  of  their 
performance  on  crowd  detection,  tracking,  and  motion  planning.  In  particular,  the  crowd 
coverage  percentage  was  considered  as  the  measure  of  effectiveness  (MOE)  in  this  work. 
An  overview  of  different  components  is  provided  in  the  following  paragraphs,  and  more 
details  on  the  proposed  framework  can  be  found  in  the  following  journal  paper: 

•  A.  Khaleghi,  D.  Xu,  Z.  Wang,  M.  Li,  A.  Lobos,  J.  Liu,  Y.  Son,  A  DDDAMS-based 
Planning  and  Control  Framework  for  Surveillance  and  Crowd  Control  via  UAVs  and 
UGVs,  Expert  Systems  with  Applications ,  40,  2013,  7168-7183. 


Decision  Module  for  DDDAMS 


Fidelity  Assignment 

Selected  Fidelity  Level 

Fidelity  Selection 

No 

^^Cat  astr  ophicaf"'" 

Yes 

Algorithm 

Algorithm 

Vabn  orm  ality 

Deviated 

Performance 


Human 

Operator 


Interruption 

task 


Figure  1:  Proposed  DDDAMS-based  planning  and  control  framework  for  surveillance 
and  crowd  control  via  UAVs  and  UGVs  (Khaleghi  et  al.,  2013) 

Integrated  Controller:  The  integrated  controller  is  in  charge  of  effective  and  efficient 
control  of  UAVs  and  UGVs,  where  the  effectiveness  is  supported  by  the  integrated  planner, 
and  the  computationally  efficiency  is  supported  by  the  decision  module  for  DDDAMS.  To 
control  UAVs  and  UGVs,  the  integrated  controller  performs  four  major  functions:  1)  crowd 
detection,  2)  crowd  tracking,  3)  motion  planning  of  UAV/UGV,  and  4)  interaction  with  the 
real  system.  To  achieve  interactions  with  the  real  system,  the  hardware  interface  in  the 
integrated  controller  acts  as  a  medium  to  collect  sensory  data  (e.g.  vision  data  and  global 
positioning  system  (GPS)  data)  from  the  real  system,  passes  them  to  the  command 
generator,  receives  control  commands  from  the  motion  planning  module,  and  sends  the 
corresponding  control  commands  to  the  real  system.  Among  received  sensory  data,  the 
vision-based  data  (e.g.  images  and  video  streams)  are  utilized  in  the  crowd  detection 


module,  and  the  GPS  data  are  used  by  the  motion  planning  module.  Given  the  vision-based 
data,  the  crowd  detection  module  1)  processes  them  for  crowd  detection  and  location 
identification  purposes,  2)  invokes  the  decision  module  for  DDDAMS  if  actual  and 
predicted  system  performances  (crowd  coverage  in  this  work)  have  a  significant  deviation, 
and  3)  passes  the  analyzed  crowd  data  (e.g.  locations  and  crowd  coverage)  to  the  database 
periodically.  Based  on  the  crowd  location  identified  at  time  stamp  t,  the  crowd  tracking 
module  predicts  the  crowd  locations  at  time  stamp  t+At,  where  At  is  an  interval  that  users 
must  define  depending  on  the  intended  frequency  of  control  (10  seconds  were  used  in  this 
work).  The  predicted  locations  are  then  used  in  the  motion  planning  module  as  destination 
locations  of  the  UAVs/UGVs,  from  which  control  commands  (e.g.  waypoints)  for  next  At 
are  generated  for  the  UAVs/UGVs.  This  process  (i.e.  data  collection  -  crowd  detection  - 
crowd  tracking  -  motion  planning  -  control  command  generation)  continues  in  an  iterative 
manner  until  the  real  system  performance  deviates  enough  from  the  predicted  performance 
(event  basis)  or  the  predefined  planning  horizon  is  elapsed  (temporal  basis),  which  then 
invokes  the  integrated  planner. 

Integrated  Planner:  The  integrated  planner,  when  invoked,  devises  an  optimal  control 
strategy  for  UAVs/UGVs  based  on  predicted  system  performance  and  passes  the  updated 
control  strategy  to  the  integrated  controller.  The  integrated  planner  in  the  proposed  work 
was  implemented  in  an  agent-based  simulation  (ABS)  environment,  where  the  strategy 
maker  selects  optimal  strategies  for  each  of  the  same  components  in  the  command 
generator  (i.e.  crowd  detection,  crowd  tracking  and  motion  planning)  based  on  simulation- 
based  evaluation  of  alternative  strategies  against  different  scenarios.  This  work  mainly 
focuses  on  1)  evaluation  of  alternative  estimation  methods  of  UAV/UGV  locations  in  At, 
and  2)  evaluation  of  multi-objective  weights  in  UAV/UGV  motion  planning.  For 
estimation  of  UAV/UGV  location,  the  crowd  shape  and  boundary  are  characterized  first 
via  clustering  technique,  followed  by  the  simulation-based  evaluation  on  UAV/UGV 
locations  contingent  to  different  control  strategies.  For  the  evaluation  of  multi-objective 
weights  in  UAV/UGV  motion  planning,  agent-based  simulation  is  used  in  the  similar  way 
for  evaluating  different  weight  combinations  of  placed  on  the  multiple  objectives  (e.g. 
shortest  travel  distance,  least  elevation  change)  over  different  scenarios.  Under  both  cases, 
each  control  strategy  only  corresponds  to  one  simulation  instance,  and  the  best  strategy  can 
be  sorted  out  (via  statistical  analysis)  after  all  simulation  instances  are  completed.  At  last, 
the  selected  best  control  strategy  is  stored  and  then  used  by  the  integrated  controller  for 
adjusting  the  corresponding  vehicle  parameters. 

Interactions  among  Components:  At  a  given  time  point  t,  when  the  decision  module  for 
DDDAMS  is  invoked,  the  checking  condition  (catastrophic  abnormality  block)  is 
processed  first.  The  checking  condition  determines  whether  the  current  control  system  has 
severe  problem  or  performance  deviations  (predicted  vs.  real)  are  too  extreme  to  recover. 
Under  these  circumstances,  the  human  operator  should  participate  for  interrupting  the  real 
system  run.  These  fatal  abnormalities  are  due  to  system  malfunctions,  human  errors,  and 
other  issues,  which  are  out  of  the  scope  of  our  analysis  in  this  work.  We  are  interested  in 
the  abnormalities,  where  the  actual  and  predicted  system  performances  deviate 
significantly  yet  every  components  still  work  in  the  normal  condition.  Under  the  ordinary 
abnormality  case,  the  fidelity  selection  algorithm  is  invoked  next.  The  outputs  of  the 


fidelity  selection  algorithm  are  a  combination  of  different  fidelity  levels  at  all  considered 
crowd  regions/cells  in  terms  of  information  details  (collected  via  UAV  or  UGV)  to  be 
incorporated  into  simulation.  In  general,  simulating  group  level  behaviors  involves  coarse 
scale  and  requires  less  information  and  computational  resource  (and  time),  while  the 
simulation  of  detailed  individual  behavior  needs  finer  scale  of  modeling,  more  detailed 
information  and  more  computational  intensive  (and  time-consuming).  Given  that  the 
deployment  of  fidelity  selection  results  in  simulation  faces  computational  constraints,  the 
fidelity  assignment  algorithm  adjusts  optimum  system  performance  against  computational 
resources  available  in  the  system.  The  fidelity  assignment  algorithm  is  formulated  as  a  0- 
1  knapsack  problem,  and  the  assigned  fidelity  levels  are  provided  to  the  integrated  planner 
and  integrated  controller  for  the  simulation  fidelity  update,  control  strategy  evaluation  and 
control  command  generation. 

Framework  Implementation:  The  proposed  planning  and  control  framework  (see  Figure 
1)  was  implemented  employing  a  number  of  state-of-the-art  software  tools:  1)  Repast 
Simphony®  (Agent-based  simulation  tool)  for  command  generator  and  strategy  maker  (see 
Figure  2),  2)  QGroundControl  for  hardware  interface.  It  is  noted  that  the  same  simulator 
used  in  the  strategy  maker  (running  in  fast-mode)  is  used  as  the  command  generator 
(running  in  real-time  mode).  To  represent  a  real  system  to  demonstrate  the  proposed 
planning  and  control  framework,  both  real  hardware  components  as  well  as  simulated 
components  were  used  under  the  real-time  hardware-in-the-loop  (HIL)  simulation 
environment  in  this  work:  3)  manually  assembled  Aducopter  and  parrot  AR.Drone  2.0  for 
UAYs  and  manually  assembled  Adurover  for  a  UGV,  4)  social  force  model  implemented 
in  Repast  Simphony®  running  in  real-time  for  simulated  crowd  behaviors,  5)  simulated 
UGVs  in  Repast  Simphony®,  6)  GIS  3D  data  from  NASA  for  simulated  environment  (i.e. 
elevation  data)  in  Repast  Simphony®,  and  7)  radio  communications  (915MHz)  by  3DR. 
More  details  on  the  hardware  components  (UAVs  and  UGVs)  and  their  integration  with 
the  real-time  HIL  simulation  will  be  described  in  Section  3.3. 


Figure  2:  Sanpshot  of  agent-based  simulation  in  contionous  3D  environment  and  GIS  3D 

projection  in  Repast  Simphony 


3.2.  Task  2:  Development  of  algorithms  to  improve  performance  of  coordinated  UAV 
and  UGVs  in  tracking  and  controlling  human  crowd 

This  section  provides  details  on  the  models  and  algorithms  that  were  developed  for  three 
major  modules  (crowd  detection,  crowd  tracking,  and  motion  planning)  pertaining  to  the 
proposed  control  framework  (see  Figure  1). 

Crowd  Detection:  To  address  crowd  detection  by  UGV  and  UAV,  we  considered  both 
real  image/stream  data  from  the  camera  in  a  real  UAV  (see  Figure  3(a))  and  from  the 
camera  in  a  real  UGV  (see  Figure  3(b))  as  well  as  models  of  them.  Key  characteristics 
considered  in  the  models  illustrated  in  Figures  3(a)  and  3(b)  are  provided  in  Table  1. 


Figure  3(a):  Illustration  of  UAV  detection  and  associated  parameters 


Figure  3(b):  Illustration  of  UGV  detection  and  associated  parameters 


Table  1:  Parameters  for  UAV  and  UGV  for  their  detection  module 


Notation 

Explanation 

Notation 

Explanation 

fovx{a) 

Horizontal  field  of  view  of  UAV’s 

camera 

FOVx{G) 

Horizontal  field  of  view  of  UGV5  s 

camera 

F0VyA- 

Vertical  field  of  view  of  UAV’s 

camera 

h{G} 

UGV’s  safety  distance 

AGL  altitude  of  UAV 

h(G) 

ILmax 

Longest  detection  distance  of  UGV’s 
camera 

DR(a) 

Detection  range  of  UAV  in  the 
horizontal  direction 

dr{G) 

Detection  range  of  UGV  in  the 
horizontal  direction 

dr(a) 

Detection  range  of  UAV  in  the 
vertical  direction 

FDD 

UGV’s  effective  detection  depth 

In  terms  of  models  of  detection  capabilities  of  UAV/UGV  (having  cameras)  during  their 
surveillance,  the  complementary  role  of  them  was  considered  in  this  research,  where  UAVs 
with  less  accurate  detection  capability  are  used  to  localize  crowd  as  groups  by  flying  over 
obstacles  to  keep  the  entire  group  in  their  FOY,  and  UGVs  are  used  for  detection  of 
individuals  in  crowd  due  to  their  accurate  localization  (see  Figure  4).  This  assumption 
considered  in  our  modeling  was  based  on  the  visual  comparison  in  Grocholsky  et  al.  (2006) 
that  has  demonstrated  uncertainties  in  localization  of  detected  objects  between  air  and 
ground  vehicles.  For  the  crowd  track  module  (discussed  in  the  next  section),  we  considered 
different  scenarios  based  on  the  data  quality  of  UAV  and  UGV  (see  Table  2). 


UGVs’  detection  renage 


UAV’s  detection  range 


(B)  UAV’s  detection 


(A)  Crowd  tracking  using 
UAV  and  UGVs 


U/  UAV 
□  UGV 


(C)  UGVs’  detection 


Figure  4:  Detection  capability  of  UAVs  and  UGVs  (Yuan  et  al.,  2015) 


In  addition  to  the  models  on  crowd  detection,  a  novel  vision-based  target  detection  and 
localization  system  was  also  developed  as  part  of  this  project  to  make  use  of  different 
detection  capabilities  of  UAVs  and  UGVs.  A  customized  motion  detection  algorithm  was 
applied  to  follow  the  crowd  from  the  moving  camera  mounted  on  the  UAV.  Due  to  UAV’s 
lower  resolution  and  broader  detection  range,  UGVs  with  higher  resolution  and  fidelity 
were  used  as  the  individual  human  detectors,  as  well  as  moving  landmarks  to  localize  the 
detected  crowds  with  unknown  independently  moving  patterns  at  each  time  point.  The 
UAV’s  localization  algorithm,  proposed  in  our  project,  then  converts  the  crowds’  image 
locations  into  their  real-world  positions,  using  perspective  transformation.  A  rule-of-thumb 
localization  method  by  a  UGV  was  also  developed,  which  estimates  the  geographic 
locations  of  the  detected  individuals.  The  algorithms  were  developed  and  customized  using 
open  source  computer  vision  libraries  (OpenCV).  Figure  5  depicts  the  detection  results  by 
a  UAV  and  UGV  for  a  scenario  of  two  moving  crowds.  More  details  on  the  vision-based 
target  detection  and  localization  system  can  be  found  in  the  following  journal  paper: 

•  S.  Minaeian,  J.  Liu,  Y.  Son,  Vision-based  Target  Detection  and  Localization  via  a  Team 
of  Cooperative  UAV  and  UGVs,  IEEE  Transactions  on  Systems,  Man,  and 
Cybernetics:  Systems  (Special  Issue  on  Biomedical  Robotics  and  Bio-mechatronics 
Systems  and  Application),  Accepted,  2015. 


(a) 


(b) 

Figure  5:  The  detection  results  for  a  scenario  of  two  moving  crowds:  (a)  detected  crowds 
by  UAV  at  two  different  time  stamps  including  UGV’s  projected  detection  range;  (b) 
detected  individuals  by  UGV  at  two  different  time  stamps  (Minaeian  et  al.,  2015) 


In  addition,  we  developed  two  camera  motion  planning  algorithms  to  enhance  the 
computational  efficiency  as  well  as  target  detection  performance.  Local  motions  necessary 
for  the  camera  to  maximize  its  visibility  of  the  human  crowd  are  considered.  It  is  noted  that 
the  camera  motion  is  to  locally  adjust  the  camera  locations/directions  to  achieve  the  best 
observations  of  the  crowd.  In  most  of  the  existing  methods  available  in  the  literature,  it  is 
assumed  that  that  maintaining  the  visibility  of  a  single  reference  point  (e.g.,  the  center  of 
mass)  will  provide  the  visibility  to  the  entire  target.  However,  it  usually  is  not  the  case  in 
the  real  world. 


Figure  6:  (a)  A  virtual  city  with  161  building;  (b)  Snapshots  of  ClOg  camera  following 

30  targets  in  the  city  environment  in  (a) 

Among  widely  known  methods,  the  Intelligent  Observer  (IO)  camera  provides  a  high 
success  rate,  but  is  extremely  inefficient.  The  Visibility- A  ware  Roadmap  (VAR)  (Oskam 
et  al.  (2009))  camera  provides  fast  online  tracking  strategy  through  the  use  of  pre-computed 
visibility  information,  but  it  performs  worse  than  IO  in  terms  of  visibility.  To  address  this 
issue,  we  propose  two  camera  planning  methods  called  Cached  Intelligent  Observers 
(CIO):  ClOg  and  CIOc  (Vo  et  al.  (2012)).  These  new  methods  provide  comparable 
performance  to  both  IO  and  VAR  while  reducing  the  offline  computation  and  maintaining 
efficiency  in  determining  camera  motions  online.  The  main  idea  is  to  incrementally  build 
and  cache  the  visibility  information  in  the  vicinity  of  the  targets  and  the  camera.  These  new 
methods  can  be  viewed  as  an  improved  IO  camera  that  reduces  the  visibility  computation 
complexity  to  almost  constant.  The  ClOg  method  begins  by  creating  a  two  dimensional 
grid.  For  each  grid  point,  ClOg  caches  a  certain  amount  of  information  about  not  only  itself 
(such  as  its  distance  from  the  nearest  obstacle),  but  also  about  other  grid  points  in  the 
network  (such  as  its  visibility  to  other  points).  Like  its  predecessor,  IO  (Becker  et  al. 
(1997)),  at  each  time  step,  ClOg  uses  prediction  and  evaluation  of  camera  and  target 
positions  to  try  and  find  an  ideal  location  for  the  camera  to  maintain  visibility  of  the  as 
much  of  the  flock  as  possible  for  the  next  time  step.  In  CIOc,  a  slightly  different  approach 
is  taken  to  storing  visibility  information  in  the  space.  First,  using  disc-like  partitions,  a 
graph  is  computed  as  well  as  a  visibility  graph  among  overlapping  partitions  of  the 
workspace.  As  with  ClOg,  these  data  structures  are  used  with  successive  cycles  of 
prediction  and  evaluation  of  future  target  and  camera  positions  to  make  decisions  about 
where  to  move  next.  Figure  6(b)  shows  an  example  of  ClOg  successfully  following  30 
targets  in  the  city  environment  shown  in  Figure  6(a). 


Crowd  Tracking:  Upon  the  detection  of  individual  agents  in  a  crowd  by  UAYs  or  UGVs, 
movement  tracking  of  these  individual  agents  is  initiated.  The  goal  is  to  predict  the  location 
of  individual  agents  at  next  time  stamp  based  on  current  and  historical  observations.  Based 
on  these  predicted  locations,  UAVs  and  UGVs  can  decide  their  optimal  control  strategy  of 
the  crowd  quantified  by  certain  performance  measure. 

In  tracking  crowds  using  both  UAVs  and  UGVs,  the  type  of  tracking  methods  can  be 
selected  based  on  the  data  quality  of  observations  from  UAVs  and  UGVs.  For  UAVs  and 
UGVs,  the  data  quality  issue  is  reflected  in  different  aspects.  For  UGVs,  the  observation 
data  are  assumed  to  be  either  error-prone  or  error-free,  depending  on  the  sensor  capability 
of  cameras  on-board.  When  the  UGVs  are  distant  from  the  crowd,  or  there  are  occlusions 
between  the  individuals  and  UGVs,  the  observations  of  individual  location  may  contain 
large  errors.  In  contrast,  when  UGVs  are  close  to  the  individuals  and  no  obstacles  block 
the  view  of  UGVs,  the  observations  from  UGVs  will  likely  to  have  ignorable  errors. 

In  our  project,  without  loss  of  generality,  observations  of  individuals  collected  from  UAVs 
were  assumed  to  have  two  levels  of  resolutions  (see  Figure  4).  When  the  altitudes  of  UAVs 
are  high,  it  is  assumed  that  UAVs  will  receive  image  data  with  low  resolution.  This  means 
UAVs  will  not  be  able  to  differentiate  individual’s  location  if  they  stand  close  to  each  other. 
In  this  sense,  UAVs  perceive  a  group  of  individuals  like  a  blob  without  further  detailed 
location  information  of  each  individual.  In  addition,  only  individuals  who  form  into  a  dense 
region  will  be  captured  by  UAVs  and  dispersed  individuals  will  not  be  captured.  A  higher 
resolution  image  can  be  obtained  when  the  altitude  of  UAVs  become  lower.  However, 
since  the  role  of  UAV  is  to  provide  an  overall  view  of  the  crowd,  altitude  of  UAVs  may 
not  be  set  low  enough  to  clearly  observe  individuals.  Therefore,  the  high  resolution 
observations  from  the  UAV  are  still  assumed  to  be  able  to  only  capture  blobs  formed  by 
dense  region.  For  either  UAV  or  UGV,  the  data  quality  issue  corresponds  to  two  scenarios, 
i.e.  high/low  resolution  observation  from  UAV  and  error-free/error-prone  observation  from 
UGVs  respectively.  These  scenarios  can  be  summarized  in  a  two-by-two  table  as  shown  in 
Table  2. 


Table  2:  Summary  of  tracking  scenarios  regarding  data  quality 


Error-free  observation 

Error-prone  observation 

(UGV) 

(UGV) 

Low  resolution  (UAV) 

Scenario  2  (current  work) 

Scenario  3 

High  resolution  (UAV) 

Scenario  4 

Scenario  1  (current  work) 

As  explained  in  Table  2,  the  current  work  considers  two  scenarios,  scenarios  1  and  2.  As 
the  methods  are  chosen  based  on  how  the  motion  dynamics  are  modeled,  the  modeling 
procedure  is  explained  first  below  followed  by  different  methods  developed  for  scenarios 
1  and  2. 

Modeling  of  crowd  motion  dynamics :  To  track  the  motion  of  individuals  in  the  crowd,  the 
model  regarding  the  motion  dynamics  of  the  individuals  needs  to  be  assumed.  In  current 
work,  the  state  space  model  is  adopted.  The  state  vector  contains  state  information  of  an 
individual  agent,  including  its  preference  state,  such  as  moving  speed  and  direction,  and 


geographic  state,  such  as  locations.  Thus  the  preference  state  vectors  are  used  as  elements 
to  model  the  complex  spatial/temporal  problem.  As  an  individual  person  in  a  civil  crowd, 
its  preference  and  geographic  states  at  time  t  are  mainly  affected  by  three  types  of  factors: 
1)  the  preference  of  crowd  individuals,  including  itself,  at  time  stamp  t- 1;  2)  the 
environmental  factors  that  affect  crowd  preferences;  and  3)  the  impacts  of  the  controlling 
units,  such  as  UAVs  and  UGVs.  Different  formats  of  the  matrixes  in  the  proposed  model 
can  be  specified  to  model  the  motion  dynamics  of  the  crowd  under  different  scenarios.  The 
common  scenarios  include  crowd  moving  towards  a  direction;  shrinking  or  spreading  of 
the  crowd,  change  of  movement  direction  and  speed,  crowd  split,  or  merge.  More  details 
on  the  proposed  model  for  crowd  motion  dynamics  can  be  found  in  the  following  journal 
paper: 

•  Yifei  Yuan,  Zhenrui  Wang,  Mingyang  Li,  Young-Jun  Son,  Jian  Liu,  DDDAS-based 
Information- Aggregation  for  Crowd  Dynamics  Modeling  with  UAVs  and  UGVs, 
Frontiers  in  Robotics  and  Al  (Sensor  Fusion  and  Machine  Perception  Section),  2:8, 
2015,  1-10. 

Tracking  Scenario  L  Tracking  scenario  1  corresponds  to  the  case  that  UAVs  observe  the 
high  resolution  data  and  UGVs  capture  individual  location  with  observation  errors.  Under 
this  scenario,  the  crowd  tracking  from  both  UAVs  and  UGVs  can  be  achieved  by  Kalman 
filter.  For  tracking  using  UGVs,  the  locations  of  all  crowd  individuals  detected  by  UGVs 
are  used  as  input.  A  Kalman  filter  is  constructed  with  a  state  vector  that  is  a  stack  up  of 
state  vectors  of  all  the  detected  individuals.  In  current  work,  it  is  assumed  that  the  values 
of  the  matrix  used  in  Kalman  filter  have  been  specified.  In  case  that  the  values  of  these 
matrices  are  unknown,  system  identification  must  be  performed  to  determine  their  values 
and  then  the  tracking  of  crowd  can  be  implemented  with  Kalman  filtering.  For  tracking 
using  UAVs,  the  location  of  crowd  center  is  estimated  from  the  collected  low  resolution 
data;  then  the  crowd  center  is  used  as  the  input  to  Kalman  filter.  As  compared  with  the 
Kalman  filter  used  in  tracking  from  UGV,  the  state  vector  used  in  tracking  from  UAV  has 
a  state  vector  of  only  four  dimensions,  i.e.  the  location  and  speed  of  the  crowd  center.  By 
tracking  the  motion  of  crowd  center,  a  highly  aggregated  information  regarding  the  motion 
of  the  crowd  is  obtained. 

Tracking  Scenario  2:  Tracking  scenario  2  corresponds  to  the  case  that  UAVs  observe  the 
low  resolution  data  and  UGVs  capture  individual  location  without  observation  errors. 
Under  this  scenario,  the  crowd  tracking  from  UAVs  cannot  be  achieved  with  Kalman  filter 
due  to  low  resolution  observation.  An  alternative  approach  to  prediction  crowd  motion 
involving  an  aggregated  model  is  proposed.  More  details  on  the  proposed  aggregation 
model  can  be  found  in  Yuan  et  al.  (2015).  Given  that  UAV  observes  low  resolution  data 
and  UGV  observes  very  accurate  information  of  certain  individual’s  location,  it  is  possible 
for  UAV  to  improve  its  prediction  by  receiving  information  aggregated  from  UGV.  Figure 
7  the  UAVs’  crowd  prediction  with  UGVs’  information  aggregation,  where  there  is  a 
discrepancy  between  predictions  by  UGV  (predicting  that  the  cell  in  the  3rd  row  and  4th 
column  will  be  occupied  by  the  crowd)  and  UAV  (predicting  that  the  same  cell  will  not  be 
occupied  by  the  crowd).  More  details  on  the  aggregation  process  can  be  found  in  Yuan  et 
al.  (2015).  As  for  tracking  using  UGVs,  when  observations  errors  are  ignorable,  the  state 
space  model  can  be  reformulated  as  an  autoregressive  (AR)  model.  Therefore,  the  tracking 


of  individuals  using  UGVs  is  achieved  by  fitting  an  AR  model.  More  details  on  the  tracking 
using  UGVs  can  be  found  in  in  Yuan  et  al.  (2015). 
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Figure  7:  Overview  of  U  A  Vs’  crowd  prediction  with  aggregated  UGVs’  information 


Motion  Planning:  In  our  project.  A*  algorithm  was  used  for  motion  planning  of 
UAVs/UGVs  to  find  their  optimal  trajectory  from  start  location  to  the  destination  location, 
where  the  destination  location  is  defined  based  on  the  output  of  the  crowd  tracking  module. 
A*  is  a  well-known  graph  search  algorithm  where  the  optimality  and  efficiency  of  chosen 
path  depends  on  the  chosen  heuristic.  It  combines  concepts  of  Dijkstra  (least  cost  from  the 
initial  location)  and  Best-First  Search  on  a  heuristic  basis  (least  cost  to  the  destination)  to 
guarantee  the  optimal  path  with  minimum  cost  in  a  reasonably  short  time  (Patal,  2013). 


Similar  to  other  graph  search  algorithms,  a  grid  is  generated  by  discretizing  the 
environment  and  a  graph  is  defined  where  each  node  of  graph  represents  one  block  of  grid. 
In  this  work,  eight-point  graph  connectivity  was  considered  to  represent  eight  different 
motions  that  UGV  can  perform.  In  addition,  it  was  assumed  the  world  is  static  and  prior 
geographical  information  about  the  environment  (e.g.  terrain  elevation)  is  known.  In  case 
of  insufficient  prior  knowledge  about  the  environment  and  existing  obstacles,  vision-based 
sensory  data  (e.g.  video  streams)  can  be  used  to  provide  such  dynamic  data  in  use.  In  our 
project,  the  terrain  information  was  obtained  by  the  GIS  data  available  in  the  NASA  world 
wind  map  (http://worldwind.arc.nasa.gov/iava/). 


In  our  project,  two  objectives  were  considered  in  motion  planning,  where  the  first  objective 
was  to  minimize  the  Euclidian  distance  of  vehicles  from  the  start  location  to  the  destination 
location,  and  the  second  objective  was  to  minimize  the  energy  consumption  of  UAV/UGV 
by  providing  penalty  for  the  paths  involving  elevation  changes.  In  addition,  the  weight  of 
each  objective  is  defined  based  on  their  relative  importance.  Figure  8  depicts  resultants 
paths  based  on  the  shortest  path,  the  least  elevation  change,  and  combination  of  both 
objectives,  respectively.  More  details  on  the  motion  planning  algorithm  can  be  found  in 
Khaleghi  et  al.  (2013). 


Figure  8:  Three  motion  planning  paths  with  minimal  travel  time,  minimal  energy 
consumption,  and  a  linear  combination  of  both  objectives 

3.3.  Task  3:  Development  of  a  hardware-in-the-loop  simulation/control  framework 
and  testbed  for  crowd  control 

This  section  provides  details  on  the  hardware-in-the-loop  simulation/control  framework 
and  testbed  for  crowd  control. 

Testbed:  To  test  the  proposed  DDDAMS  planning  and  control  framework  (see  Figure  1), 
a  testbed  representing  a  real  system  (3  UAYs,  1  UGV,  human  crowd,  environment  such  as 
terrain  and  location)  was  developed.  The  testbed  in  our  project  is  composed  of  both  real 
hardware  components  as  well  as  simulated  components:  1)  manually  assembled  Aducopter 
(UAV),  parrot  AR.Drone  2.0  (UAV),  X8+  UAV  by  3DR,  and  manually  assembled 
Adurover  (see  Figure  9),  2)  simulated  crowd  and  simulated  UGVs  implemented  in  Repast 
Simphony®  agent-based  simulation,  3)  GIS  3D  data  from  NASA  for  simulated 
environment  (i.e.  elevation  data)  in  Repast  Simphony®,  and  4)  radio  communications 
(915MHz)  by  3DR. 


Figure  9:  Aducopter,  Parrot  AR.  Drone  2.0,  and  Adurover  used  in  the  testbed 

To  accomplish  the  testbed  involving  both  hardware  components  as  well  as  software 
components,  we  developed  hardware-in-the-loop  (HIL)  real-time  simulation  platform 
using  Repast  Simphony®  agent-based  simulation  tool.  To  provide  a  communication 
network  between  the  ground  control  station  and  the  vehicles  (see  Figure  10),  we  evaluated 
various  technologies.  3DR®  Radio  developed  by  3dRobotics®  (http://3drobotics.com/) 
was  our  first  choice,  which  can  provide  915  MHz  radio  communication  with  air  data  rates 
up  to  250  kbps  and  approximate  extended  range  of  1  mile.  This  radio  module  was  specially 
designed  to  transmit  telemetry  data  using  MAVLink®  (Micro  Air  Vehicle  Link). 


MAVLink®  (http://www.qgroundcontrol.org/mavlink/start)  is  an  open  source  protocol  for 
communication  between  the  ground  control  station  and  unmanned  vehicles,  which 
provides  a  standard  message  format  for  sending  waypoints.  It  is  noted  that  the  same 
message  format  is  generated  using  the  agent-based  simulation  of  our  testbed  as  the  control 
commands.  More  details  of  the  MAVLink®  message  format  can  be  found  in 
http://mavlink.org/messages/common.  While  3DR®  radio  module  has  been  used  in  a 
variety  of  unmanned  vehicles  to  provide  a  one-to-one  communication  network  between  a 
vehicle  and  the  ground  control  station,  a  network  is  required  in  our  testbed  with  a  hybrid 
control  architecture  to  provide  communication  between  multiple  vehicles  and  the  ground 
control  station.  In  this  regard,  our  second  choice  of  communication  network  in  our  project 
was  Xbee®  Pro  900  by  Digi®.  Even  though  this  module  has  the  capability  of  providing  a 
mesh  network,  it  has  not  been  designed  for  working  with  MAVLink®,  which  causes  a 
delayed  communication  in  transmitting  telemetry  data.  To  overcome  this  challenge,  we 
selected  Wi-Fi  using  MAVLink®  as  our  third  choice  of  the  communication  network  in  the 
project. 


Sensory  Dat; 
(e.g.  GPS) 


I  Hardware  Interface 


Wi-Fi /XBee  PRO  900HP 


Control  Commands 
(MAVLink  Messages) 

Assembled  UGV 
(APM :  Rover  /  Ard  u  rover) 


Figure  10:  Hardware-in-the-loop  integration 

UAVs  (Aducopter  and  Parrot  AR.  Drone  2.0):  This  section  provides  details  on  three 
types  of  UAVs  (Aducopter,  Parrot  AR.  Drone  2.0,  X8+  from  3DR)  that  were  acquitted  to 
develop  the  proposed  testbed. 


Aducopter  and  Adurover  ( see  Figure  9):  The  hardware  for  the  aircraft  including  battery 
and  autopilot  system  costs  less  than  $300  USD  and  is  modular,  consisting  entirely  of  off- 


the-shelf  and  open-source  parts.  The  airframe  is  a  customized  quad-rotor  helicopter  with  a 
fiberglass  body  and  removable  ABS  plastic  arms.  With  a  diameter  of  450mm  and  weight 
of  800g  without  battery  installed  (1200g  with  a  5000  mAh  lithium  polymer  battery 
installed),  this  small  UAV  can  be  operated  indoors  safely  and  navigate  small  spaces.  Each 
arm  is  fitted  with  a  brushless  outrunner  direct-current  motor  (RCTimer  HP2217)  with  10- 
inch  diameter  carbon  fiber  fixed-pitch  propellers  installed.  Each  motor  is  controlled  by  an 
electronic  speed  control  (ESC)  circuit  (RCTimer  40A  OPTO  ESC)  that  accepts  speed 
commands  at  490  Hz  from  an  autopilot  module  called  the  ArduPilot  Mega  2.5  (APM  2.5). 
The  ArduPilot  is  an  open-source  aircraft  control  board  based  on  ATmega  2560 
microprocessor.  It  handles  input  from  the  on-board  sensors  (3-axis  gyro,  3-axis 
accelerometer,  3 -axis  magnetometer,  barometer,  and  GPS  unit)  and  sends  pulse-width- 
modulation  (PWM)  signals  to  the  ESCs.  By  itself,  it  can  perform  automated  stabilization, 
GPS-based  autonomous  navigation,  and  radio  control  input  from  a  2.4GHz  band  handheld 
transmitter.  However,  for  this  project,  we  are  interested  in  controlling  the  aircraft  using  our 
own  algorithms.  Therefore,  we  will  use  the  autopilot  board  for  low  level  control  of  the 
aircraft,  and  command  it  via  serial  (MAVLink  protocol)  from  a  quad-core  ARM  Cortex- 
A9  embedded  PC  on-board.  This  on-board  PC  will  process  the  images  captured  via  on¬ 
board  camera  and  then  send  commands  to  the  autopilot  to  execute.  For  hardware-in-the- 
loop  operation,  (i.e.  in  the  absence  of  the  embedded  PC),  the  ArduPilot  may  also  accept 
commands  wirelessly  using  a  ISM-band  (915  Mhz)  radio  receiver  on-board.  An  Adurover 
(see  Figure  9)  was  custom  built  in  a  similar  manner  as  well. 

Parrot  AR.  Drone  2.0  (see  Figure  9):  Parrot  AR.  Drone  2.0  is  equipped  with  1)  a  front 
camera  (CMOS  sensor  with  a  90  degrees  angle  lens,  video  frequency  of  30  fps,  resolution 
of  1280*720  pixels),  2)  ultrasound  sensor  (emission  frequency  of  40  kHz  and  range  with  6 
meters),  3)  embedded  computer  system  (CPU  OMAP  3630  1GHz  ARM  cortex  A8,  DDR 
SDRAM  128MB,  NAND  Flash  memory  128M,  Wi-Fi  /g/n,  Linux  OS),  and  4)  battery 
(Lithium  polymer  battery  (3  cells,  11.1V,  1000  mAh;  Charging  time:  1.5  hours;  running 
time  of  12  minutes).  With  WIFI  N,  it  can  fly  as  far  as  165  feet,  and  it  is  controllable  from 
any  client  device  supporting  Wi-Fi  Communication.  It  uses  different  ports:  UDP  port  5556 
for  controlling  and  configuring,  TCP  port  5555  for  video  stream,  and,  TCP  port  5559 
(control  port)  to  transfer  critical  data,  by  opposition  to  the  other  data  that  can  be  lost  with 
no  dangerous  effect. 


X8±  UAV  by  3DR  (see  Figure  9):  X8+  from  3DR  is  equipped  with  1)  a  controller  with  live 
on-screen  flight  data,  2)  flight  battery,  3)  8  APC  propellers  (4  SF  and  4  SFP),  4)  3DR  u- 
blox  GPS  with  Compass,  5)  Ground  station  radio  (3DR  Radio  v2  -915  MHz),  and  6) 
Advanced  Pixhawk  v2.4.5  autopilot  system.  Some  major  features  include  1)  maximum 
altitude  of  100  m  (328  ft),  2)  range  (300  m  (984  ft)  from  launch  point),  3)  max  flight  time 
of  15  minutes,  4)  payload  capacity  of  800  g  (1.7  lbs),  5)  GPS  lock  required  at  all  times,  6) 
autonomy  by  a  Mac,  PC  or  Android  device,  7)  smooth  and  reliable  flight  dynamics,  8) 
flight  Protection(land  itself  automatically,  or  return  to  launch),  9)  automatic  mission 
planning  software,  and  10)  compatible  with  gimbal  stabilizer. 


Section  4:  Accomplishments  /  New  Findings:  Research 
Highlights 

This  section  provides  details  on  the  results  and  accomplishments  that  we  have  obtained  for 
three  major  tasks  (see  Section  2)  based  on  the  approach  and  detailed  methods  discussed  in 
Section  3. 

4.1.  Task  1:  Development  and  refinement  of  coherent  planning  and  control 
framework  (DDDAMS)  for  effective  and  efficient  control  of  UAV/UGVs  (Ta) 

The  proposed  DDDAMS  framework  (see  Figure  1)  was  successfully  demonstrated  for  the 
testbed  based  on  agent-based  real-time  simulation  (including  three  types  of  real  UAVs,  one 
real  UGV,  radio  communications,  simulated  UGVs/UGVs,  real  crowd,  simulated  crowd, 
and  simulated  environment)  (see  Figure  10).  In  addition,  as  mentioned  in  Section  2,  the 
research  outcomes  (e.g.  DDDAMS  framework,  information  aggregation  framework, 
underlying  algorithms,  and  experiments  involving  simulation-based  testbed)  developed  in 
this  project  have  been  published  in  multiple  journals  and  conference  proceedings  (see 
Section  6  for  a  complete  list  of  publications). 

4.2.  Task  2:  Development  of  algorithms  to  improve  performance  of  coordinated  UAV 
and  UGVs  in  tracking  and  controlling  human  crowd 

This  section  provides  details  on  the  numerical  simulation  studies  that  were  conducted  for 
two  scenarios  (see  Table  2)  to  evaluate  the  performance  of  the  developed  crowd  tracking 
and  motion  planning  methods. 

Scenario  1:  As  mentioned  earlier,  scenario  1  corresponds  to  the  case  that  UAVs  observe 
the  high  resolution  data  and  UGVs  capture  individual  location  with  observation  errors.  In 
this  scenario,  a  case  study  was  conducted  for  a  hypothetical  surveillance  and  crowd  control 
scenario  considering  two  different  fidelities,  where  in  low  fidelity  two  UAVs  perform 
tracking  by  observing  crowd  as  group  and  in  high  fidelity  tracking  has  been  done  using 
two  UGVs  by  observing  all  individuals  in  the  crowd.  A  border  environment  in  the  state  of 
Arizona  in  U.S.  was  selected  and  a  crowd  of  80  individuals  in  two  separate  groups  were 
considered  in  Repast  Simphony,  where  their  comfortable  walking  speed  was  assumed  to 
be  around  1.5  m/s.  Crowd  locations  were  observed  in  each  1  time  stamp  in  simulation. 
The  crowd  tracking  module  utilized  these  observations  and  predicted  the  crowd  locations 
after  10  time  stamps  in  both  fidelities.  In  the  motion  planning  module,  the  high  and  low 
simulation  fidelities  utilize  the  predicted  locations  from  UGV  and  UAV,  respectively. 
Furthermore,  different  spatial  resolutions  (grid  sizes)  have  been  considered  to  perform 
motion  planning  in  high  and  low  fidelities  due  to  different  specifications  of  real  hardware. 

Figure  1 1  demonstrates  the  performance  of  the  two  fidelity  scenarios  as  crowd  coverage 
percentage  in  intervals  of  10  time  stamps.  As  seen  in  the  figure,  high  fidelity  has  higher 
performance  due  to  the  utilization  of  individual  observations  in  tracking  module  as  well  as 
more  accurate  locations  of  vehicle  motion  planning  under  finer  grid.  However,  more 
system  computational  resources  are  needed  in  the  high  fidelity.  Figure  12  illustrates  CPU 


usages  of  the  tracking  module  under  situations  with  the  high  and  low  simulation  fidelity, 
and  such  difference  mainly  lies  in  the  dimension  of  state  vectors  used  in  the  tracking 
algorithm.  In  the  case  with  low  fidelity,  the  center  of  the  crowd  dense  region  was  collected 
and  inputted  to  the  tracking  algorithm,  which  leads  to  a  state  vector  of  dimension  of  4.  In 
contrast,  under  the  case  with  high  simulation  fidelity,  under  which  every  individual  agent 
was  tracked  in  the  crowd  tracking  algorithm,  the  state  vector  has  a  dimension  that  is  a 
product  between  4  and  the  total  number  of  individual  agents.  For  our  experiments  with  40 
individuals  in  the  crowd,  the  state  vector  has  a  dimension  of  160,  which  consumed  more 
computations  during  the  tracking. 


Time  (second) 

•  Low  Fidelity 


Figure  12:  CPU  usage  for  tracking  module  under  two  simulation  fidelity  cases  with  95% 

confidence  interval 

As  Figure  12  shows  from  time  stamps  10  to  120,  the  two  fidelities  result  in  similar 
performance,  which  justifies  the  low  fidelity  case  can  maintain  similar  system  performance 


with  high  fidelity  case  under  certain  circumstances.  Then,  a  hypothetical  event  (i.e.  some 
people  start  to  lead  the  entire  group)  was  purposely  created  at  time  stamp  70,  which  causes 
4  individuals  in  each  group  increase  their  comfortable  walking  speed  from  1.5  m/s  to  1.7 
m/s.  As  the  results  show,  due  to  this  event  the  low  fidelities  performance  has  been 
decreased  dramatically  after  some  delay  (this  delay  is  the  time  for  the 
leaders  separating  from  the  major  group).  As  the  result  shows,  the  performance  in  high 
fidelity  has  also  been  reduced,  but  compared  with  that  of  the  case  with  low  fidelity,  the 
effect  of  the  event  is  much  less  due  to  tracking  individuals  instead  of  that  of  crowd.  By 
adopting  high  fidelity,  the  required  performance  (80%  coverage  percentage)  is  maintained, 
but  more  computational  resources  are  consumed. 

Scenario  2:  As  mentioned  earlier,  scenario  2  corresponds  to  the  case  that  UAVs  observe 
the  low  resolution  data  and  UGVs  capture  individual  location  without  observation  errors. 
In  this  scenario,  fifty  individuals  with  changing  motion  dynamics  are  simulated  within  a 
time  duration  of  15  time  instances.  The  fifty  individuals  are  monitored  by  one  UAV  and 
two  UGVs.  The  UAV  observes  the  overall  crowd,  and  obtains  low  resolution  data  that  only 
capture  the  dense  region.  Each  UGV  monitors  one  subgroup  of  ten  individuals.  The  motion 
dynamics  of  the  crowd  is  illustrated  in  Figure  13.  The  figure  shows  that  from  time  1  to  4, 
all  fifty  individuals  (denoted  as  black  solid  dots)  move  towards  the  right.  The  blue  curves 
represent  the  detection  range  of  two  UGVs.  From  time  5  to  6,  two  subgroups  of  individuals 
leave  the  majority  of  the  crowd.  One  subgroup  moves  towards  the  northeast  direction  and 
the  other  subgroup  moves  towards  the  southeast  direction.  From  time  7  to  15,  the  subgroup 
in  the  northeast  comer  of  the  crowd  keeps  moving  toward  to  the  right.  The  subgroup  in  the 
southeast  corner  of  the  crowd  concentrates  and  a  dense  region  is  formed.  Before 
concentration,  the  subgroup  has  low  density  and  UAV  fails  to  observe  these  data. 
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Figure  13:  Motion  dynamics  at  different  time  instances 


The  proposed  crowd  motion  prediction  approach  is  adopted  in  predicting  the  motion  of 
dense  region  captured  by  UAV.  The  observations  of  individuals  from  UGV  are  assumed 
error-free  and  autoregressive  model  is  used  to  estimate  motion  dynamics  and  predict  future 
locations.  Given  the  predicted  location  of  dense  region  and  individuals  in  two  subgroups, 
two  prediction  outcomes  are  considered.  The  first  one  is  solely  based  on  the  UAV 
prediction  and  the  second  one  combines  prediction  outcomes  from  both  UAV  and  UGV 
through  information  aggregation. 
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Figure  14:  Tracking  results  illustration  at  time  8 

The  example  tracking  results  are  illustrated  in  Figure  14  and  Figure  15.  The  first  panel  in 
Figure  14  shows  the  location  of  individuals  at  time  8,  where  the  dense  regions  with  more 
than  10  individuals  in  the  cell  are  illustrated  with  black  filled  square.  The  second  panel 
shows  the  prediction  solely  based  on  UAV.  The  true  locations  of  individuals  are  plotted. 
The  predicted  locations  are  illustrated  with  gray  squares.  Two  subgroups  of  individuals, 
each  forming  a  dense  region,  are  not  correctly  predicted  by  UAV.  The  cells  corresponding 
to  these  two  subgroups  of  individuals  are  highlighted  with  blank  squares  having  red  edges. 
The  third  panel  shows  the  combined  prediction.  To  emphasize  the  benefit  of  using 
combined  prediction,  the  extra  cells  added  in  combined  prediction  that  correctly  predicts 
two  subgroups  are  highlighted  in  green  squares.  The  tracking  performance  is  evaluated  as 
the  coverage  value  (i.e.,  proportion  of  individuals  captured  by  UAV  and  UGVs).  The 
coverage  of  using  UAV  alone  is  compared  with  combined  prediction.  The  right  panel 
shown  in  Figure  14  shows  the  coverage  value  up  to  time  8. 

Figure  15  shows  the  snapshot  of  tracking  outcome  at  time  15.  Again  both  the  tracking 
illustration  and  the  coverage  value  indicate  that  using  combined  prediction  outperforms 
prediction  based  on  UAV  alone.  This  shows  the  effectiveness  of  the  proposed  information 
aggregation  approach  in  improving  the  tracking  performance. 
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Figure  15:  Tracking  results  illustration  at  time  15 

4.3.  Task  3:  Development  of  a  hardware-in-the-loop  simulation/control  framework 
and  testbed  for  crowd  control 


Flight  Test  of  UAVs:  Both  the  UAVs  (Aducopter  and  Parrot  AR.  Drone  2.0  in  Figure  9) 
and  UGV  (Adurover  in  Figure  9)  were  successfully  tested  in  the  field.  Our  tests  revealed 
that  Aducopte  was  able  to  remain  stable  while  carrying  an  additional  payload  of  up  to 
1200g,  and  can  remain  in  the  air  for  up  to  30  minutes  with  one  5000  mAh  11. IV  lithium 
polymer  battery.  Using  the  GPS  unit  on-board  and  the  autopilot  unit,  the  UAV  can 
autonomously  navigate  to  given  waypoints  outdoors  with  an  accuracy  of  about  +/-  2  meters 
despite  wind  and  other  environmental  influences.  With  the  addition  of  a  high-accuracy 
sonar  unit,  the  UAV  can  hold  altitude  and  position  with  centimeter  accuracy  and 
autonomously  takeoff  and  land.  Live  telemetry  data  from  the  UAV  including  GPS  position, 
compass  orientation,  and  pitch/roll  orientation  is  streamed  to  a  ground  station  PC  where  it 
can  be  viewed  online  or  saved  to  a  log  file.  The  test  with  Parrot  AR.  Drone  2.0  and 
Adurover  provided  similar  performance  results.  Figure  16(b)  shows  a  snapshot  of  recorded 
path  (a  series  of  waypoints)  from  Google  map,  where  physical  flights  based  on  the  provided 
waypoints  were  successfully  demonstrated. 

Testbed:  The  proposed  testbed  based  on  agent-based  real-time  simulation  (including  three 
types  of  real  UAVs,  one  real  UGV,  radio  communications,  simulated  UGVs/UGVs, 
simulated  crowd,  and  simulated  environment)  (see  Figure  10)  were  successfully 
established.  Figure  16(a)  shows  a  snapshot  of  demo  of  the  proposed  hardware-in-the-loop 
testbed.  The  constructed  testbed  is  modular,  so  additional  hardware  equipment  and/or 
software  modules  can  be  appended  to  the  current  testbed  easily. 


Figure  16:  Snapshot  of  demo  of  Hardware-in-the-loop  testbed;  snapshot  of  recorded  path 

from  Google  map  (GPS  waypoint  flights) 
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Changes  in  research  objectives  (if  any): 

NA 

Change  in  AFOSR  Program  Manager,  if  any: 

NA 

Extensions  granted  or  milestones  slipped,  if  any: 

The  original  project  end  date  was  April  30,  2015,  which  was  in  the  middle  of  Spring  201 5.  And,  the 
Research  Assistantship  (RA)  contracts  for  the  graduate  students  continued  till  May,  2015.  So,  no  cost 
extension  was  requested  till  Sep.  30,  2015,  and  it  was  granted.  The  remaining  balance  was  used  to 
continue  to  support  graduate  students  for  the  rest  of  Spring  201 5  RA  contract  as  well  as  summer  2015.  The 
project  scope  remained  unchanged.  During  the  extended  project  periods,  graduate  students  under  the 
guidance  of  Pis  refined  the  testbed  that  the  project  team  had  developed,  and  conducted  experiments  using 
the  testbed. 
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